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KEY TAKEAWAYS 

Online Testing Is Poised For A Major Expansion Beyond On-Site 
Experimentation 

Online testing programs must follow the ongoing evolution of digital marketing 
to support multiple channels such as social and mobile and to optimize the full 
customer experience. 

Vendors Are In Rapid Development Mode To Keep Pace With Growing 
User Needs 

Online testing platform vendors are responding to expanding user requirements 
with support for multichannel testing, visitor profiling, advanced testing scenarios, 
and automation features. 

Leaders Provide Broad Online Testing Functionality 

All of the evaluated online testing platform vendors offer foundational campaign 
management and testing features, but Leaders provide extensive data handling, 
reporting and analysis, and integration capabilities. 
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WHY READ THIS REPORT 

In Forrester's 53-criteria evaluation of online testing platform vendors, we identified the seven most 
significant providers — Adobe, Autonomy, Maxymiser, Monetate, Optimizely, SiteSpect, and Webtrends — 
in the category and researched, analyzed, and scored them. This report details our findings about how well 
each vendor fulfills our criteria, and where these vendors stand in relation to each other, to help customer 
intelligence professionals select the right partner for their online testing needs. 
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OPTIMIZATION IS A CORE COMPETENCY FOR DIGITAL MARKETERS 

As customer interactions spread across myriad touchpoints and sophisticated consumers demand 
seamless and relevant experiences, marketers have been forced to reevaluate their strategies for 
engaging customers. 1 But the complexity, pace, and volume of modern digital marketing easily 
overwhelm traditional planning and design approaches that rely on historical conventions, myopic 
single- channel perspectives, and sequential act-and-learn iteration. How does online testing help? 
Online testing applies statistical techniques to compare how variations of customer experiences 
perform against a control group to determine the best treatment option. Online testing is appealing 
not only because it is efficient and measureable but also because it cuts through noise and 
assumptions to help marketers present the most effective content, promotions, and experiences to 
customers and prospects. 

Maturing Vendor Solutions Offer A Glimpse Of The Future Of Online Testing 

The evolving digital marketing landscape drives expanding requirements for online testing: to 
operate in more channels, handle more data, and support more users. Online testing must move 
beyond traditional on-site experimentation to fully optimize todays multifaceted digital customer 
experiences. This vision is entirely achievable because the underlying techniques of online testing 
apply to any channel or event that can be measured. In response, vendors are moving beyond 
standard online testing functionality — the design, execution, and measurement of experiments on 
websites — to deliver next-generation optimization capabilities: 

■ Distributed access. Supporting diverse — and increasingly nontechnical — participants in the 
online testing process requires increased emphasis on application ease of use. 2 A high degree 
of usability encourages increased testing activity by making complex functionality efficient and 
accessible. Online testing platforms also need enhanced administration, collaboration, and 
workflow functionality to support rigorous online testing processes. 

■ Multichannel testing. As customer experiences extend beyond the website to incorporate 
emerging channels and as marketers work to master new devices, usage patterns, and 
interaction types, the need for optimization is greater than ever. Online testing platforms 
must follow suit to support mobile websites, applications, social media, and rich media, each 
requiring its own experiment design, execution, and measurement capabilities. 

■ Visitor profiling. Visitor data is the common denominator between marketing and analytics 
applications as organizations track, analyze, and design multichannel customer interactions. 3 
Online testing platforms must evolve beyond siloed test management tools by incorporating the 
ability to collect, store, and distribute visitor data. Enhanced data support will allow users to target 
granular audiences, generate advanced experiments, and calculate relevant success metrics. 
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■ Advanced testing scenarios. Marketers require online testing platforms to support experiments 
that mirror complex user experiences. Online testing platforms must advance past swapping 
out HTML copy and images to running experiments spanning multiple pages and domains, 
dynamically modifying code, and testing functional components. Users also need the ability to 
calculate sophisticated success metrics such as scored and continuous measurements. 

■ Smarter optimization. Managing an online testing program puts tremendous pressure on 
users that are already time- and resource-constrained. Online testing platforms must automate 
features to make testing simpler and more effective, and they need to automate core testing 
processes, such as the construction, validation, and execution of experiments. They must also 
automate analysis with proactive recommendations and segment discovery. 

■ Optimization suites. Online testing is part of a broad spectrum of optimization techniques. 
Online testing platforms must bring these capabilities — such as rule- and algorithm-based 
targeting, recommendations, and A/B and multivariate testing — together into a single offering. 
An integrated optimization suite enhances consistency and efficiency through a one-time 
implementation, a unified data layer, and single integrations with analytics and marketing tools. 

ONLINE TESTING PLATFORM EVALUATION OVERVIEW 

To assess the state of the online testing platform market and see how the vendors stack up against 
each other, Forrester evaluated the strengths and weaknesses of top online testing platform vendors. 

Evaluation Criteria Focus On Growing Online Testing Requirements 

After examining past research, user need assessments, and vendor and expert interviews, we 
developed a comprehensive set of evaluation criteria. We evaluated vendors against 53 criteria, 
which we grouped into three high-level buckets: 

■ Current offering. To assess product strength, we evaluated each offering against seven 
dimensions: campaign management; application usability and administration; data handling; 
testing; reporting and analysis; integration; and service and support. 

■ Strategy. We compared the product and corporate strategies of each vendor in the context of 
Forrester's forward-looking vision for the online testing platform market to assess how well each 
vendor is positioned for future success. We also incorporated quantitative client feedback as 
indicated by customer satisfaction and Net Promoter Score." 

■ Market presence. To benchmark the relative size of the vendors' online testing businesses, we 
evaluated each company's financial strength, installed base of clients, and employee base. We 
paid particular attention to the revenue, clients, and staff that are specifically related to the 
vendors' online testing businesses. 
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Evaluated Vendors Have The Product Offering And Focus To Support Enterprise 
Clients 

Forrester included seven vendors in the assessment: Adobe, Autonomy, Maxymiser, Monetate, 
Optimizely, SiteSpect, and Webtrends. Each of these vendors (see Figure 1): 

■ Supports a range of online testing techniques. To meet the varied and long-term requirements 
of online testing users, all vendors that made the cut offer applications that support a range of 
testing techniques. Specifically, each evaluated vendor offers both A/B and multivariate testing 
techniques or similar equivalents. Vendors that support only A/B testing were excluded. 

■ Boasts an established customer base. We sought to include vendors with an established 
presence in the market and a demonstrable track record with live client deployments. All 
evaluated vendors offer an online testing application that is currently generally available, with 
50 or more customers in live production as of October 2012. 

■ Has a significant base of enterprise-class clients. This Forrester Wave™ online testing 
evaluation is designed to identify vendors that meet the demanding requirements of enterprise- 
class businesses, defined as firms with at least $1 billion in revenue or 1,000 employees. Each 
vendor evaluated has at least 25 enterprise-class customers in live production, comprising a 
minimum of 10% of its total client base. 

ONLINE TESTING VENDORS STRAIN TO BALANCE FUNCTIONALITY AND USABILITY 

The evaluation uncovered a market in which (see Figure 2): 

■ Maxymiser, Adobe, Autonomy, and SiteSpect lead the market. Adobe and Autonomy 
both claimed leadership positions in Forrester's first online testing platform Wave and have 
maintained those positions with a combination of strong product offerings, solid strategy, and 
a substantial market presence. 5 Maxymiser and SiteSpect — both former Strong Performers — 
make the jump to the Leader segment this year through incremental product enhancements, 
enhanced visions for optimization, and improved client reference quality. The online testing 
platform Leaders generally favor deep functionality over usability, although user experience 
design is an area of increasing priority in their product road maps. 

■ Webtrends, Monetate, and Optimizely offer competitive options. Webtrends maintains its 
position as a Strong Performer this year with highly robust testing capabilities. Monetate and 
Optimizely are new entrants in this year's Forrester Wave and are a breath of fresh air in the 
online testing platform market. Monetate succeeds with deep productization for eCommerce, 
and Optimizely maintains a laser focus on reducing the complexity of online testing for 
businesses of all sizes. Both Monetate and Optimizely are most notable for their commitment to 
application usability and deserve credit for bringing legitimate innovation to online testing user 
interface design. 
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This evaluation of the online testing platform market is intended to be a starting point only. We 
encourage clients to view detailed product evaluations and adapt criteria weightings to fit their 
individual needs through the Forrester Wave Excel-based vendor comparison tool. 



Figure 1 Evaluated Vendors: Product Information And Selection Criteria 
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Q4 2012 
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Q4 2012 
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Q4 2012 
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Q4 2012 


SiteSpect 
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Webtrends 


Optimize 
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Inclusion criteria 

Each vendor evaluated offers both A/B and multivariate testing techniques or similar equivalents. 

Each vendor evaluated offers an online testing application that is currently generally available, with 50 or 
more customers in live production as of October 2012. 

Each vendor evaluated has at least 25 enterprise-class customers in live production, comprising a 
minimum of 10% of its total client base. 
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Source: Forrester Research, Inc. 
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Figure 2 Forrester Wave™: Online Testing Platforms, Ql '13 
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Figure 2 Forrester Wave™: Online Testing Platforms, Ql '13 (Cont.) 
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Source: Forrester Research, Inc. 



VENDOR PROFILES 

Leaders 

■ Maxymiser. Maxymiser was founded in the UK but exhibits a strong commitment to the US 
market, having gone so far as to relocate its executive management team to New York. MaxTEST 
exhibits consistently strong capabilities across all categories of online testing functionality, with 
particular depth in reporting and analysis. The company has also been aggressive in supporting 
testing in mobile and social channels. Maxymiser will appeal most to users that require a 
balance of professional services support and advanced online testing technology. To retain its 
leadership position, Maxymiser must execute on its vision for comprehensive optimization and 
visitor data management. 

■ Adobe. Test&Target is a component of Adobe Marketing Cloud, which incorporates a broad set 
of analytics, content management, and marketing applications. Test&Target is a solid product 
across all facets of online testing capability and boasts an outstanding partner ecosystem. It will 
appeal most to current Adobe users for its integration with popular Adobe analytics and content 
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management products. To retain its leadership position, Adobe must improve the ease of 
implementing and maintaining Test&Target and must accelerate product development to keep 
pace with more nimble competitors. 

■ Autonomy. Optimost is a component of Autonomy's Promote product line, which is based on 
the company's Intelligent Data Operating Layer (IDOL) analytics platform. Autonomy offers 
strong testing capabilities featuring a variety of optimized techniques and will appeal most to 
users with extremely advanced integration, administration, and workflow requirements. To 
retain its leadership position, Autonomy must overhaul its badly dated user interface, which lags 
the rest of the market by a significant margin, and address ongoing market uncertainty relating 
to its 2011 acquisition by HP. 

■ SiteSpect. SiteSpect is often associated with its unique tagless deployment model, which creates 
fundamental — and polarizing — differentiation from every other vendor in the market. This 
preconception does a disservice to users, because SiteSpect offers very strong testing capabilities. 
SiteSpect will appeal most to technical users that need to balance advanced testing capabilities 
with deployment concerns stemming from technical, security, or performance requirements. 

To retain its leadership position, SiteSpect must accelerate growth by improving application 
usability and doubling down on the benefits of its nonintrusive deployment model, particularly 
for testing in emerging channels such as mobile. 

Strong Performers 

■ Webtrends. Optimize is a component of Webtrends' platform covering social, mobile, and web 
analytics. At first blush, Optimize seems less flexible than its peers, but this is a feature rather 
than a limitation. Webtrends applies a rigorous approach to experiment design, execution, and 
analysis, with the intention of providing the most statistically trustworthy approach possible. 
Webtrends will appeal most to users with advanced online testing requirements delivered via 
managed services. To remain competitive, Webtrends must reduce its reliance on professional 
services supported engagements with a viable self-service offering and integrate Optimize with 
its most innovative new suite offerings, such as the real-time Streams product. 

■ Monetate. Monetate has enjoyed a meteoric rise in the past two years, evolving from a startup 
into a legitimate alternative for enterprise online testing. The company is well funded and 
markets itself aggressively, and the product provides strong testing and targeting functionality 
in an easy-to-deploy and easy-to-use package. Monetate provides a highly productized offering 
for eCommerce and multichannel retailers and has exceptionally high client- satisfaction scores 
to show for it. To remain competitive, Monetate must expand its industry footprint, continue 
developing advanced testing features, and expand its support and services resources to keep up 
with customer growth. 
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■ Optimizely. Optimizely is the newest entrant to the market for enterprise online testing. From 
humble beginnings as a cheap, no-frills A/B testing tool for the small and medium-size business 
(SMB) market, Optimizely now serves a substantial number of enterprises by addressing the 
needs of marketing-oriented users that have historically struggled to achieve testing success 
with more complex and costly tools. The company follows a strict ethos of simplicity as a 
method of democratizing online testing, and its most notable differentiator is a modern and 
highly intuitive user interface. To remain competitive, Optimizely must further establish its 
enterprise credibility, expand its professional services and support capabilities, and apply its 
innovative UX designs to advanced testing features. 

SUPPLEMENTAL MATERIAL 
Online Resource 

The online version of Figure 2 is an Excel -based vendor comparison tool that provides detailed 
product evaluations and customizable rankings. 

Data Sources Used In This Forrester Wave 

Forrester used a combination of four data sources to assess the strengths and weaknesses of each 
solution: 

■ Vendor surveys. Forrester surveyed vendors on their capabilities as they relate to the evaluation 
criteria. Once we analyzed the completed vendor surveys, we conducted vendor calls where 
necessary to gather details of vendor qualifications. 

■ Product demos. We asked vendors to conduct demonstrations of their product's functionality. We 
used findings from these product demos to validate details of each vendor's product capabilities. 

■ Customer reference calls. To validate product and vendor qualifications, Forrester also 
conducted reference calls with three of each vendor's current customers. 

■ Customer reference online survey. To establish a level of customer satisfaction across a number 
of categories including product functionality, flexibility, usability, services, and support, Forrester 
also conducted an online survey of a minimum of 15 of each vendor's current customers. 

The Forrester Wave Methodology 

We conduct primary research to develop a list of vendors that meet our criteria to be evaluated 
in this market. From that initial pool of vendors, we then narrow our final list. We choose these 
vendors based on: 1) product fit; 2) customer success; and 3) Forrester client demand. We eliminate 
vendors that have limited customer references and products that don't fit the scope of our evaluation. 
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After examining past research, user need assessments, and vendor and expert interviews, we develop 
the initial evaluation criteria. To evaluate the vendors and their products against our set of criteria, 
we gather details of product qualifications through a combination of lab evaluations, questionnaires, 
demos, and/or discussions with client references. We send evaluations to the vendors for their review, 
and we adjust the evaluations to provide the most accurate view of vendor offerings and strategies. 

We set default weightings to reflect our analysis of the needs of large user companies — and/ or 
other scenarios as outlined in the Forrester Wave document — and then score the vendors based 
on a clearly defined scale. These default weightings are intended only as a starting point, and we 
encourage readers to adapt the weightings to fit their individual needs through the Excel-based 
tool. The final scores generate the graphical depiction of the market based on current offering, 
strategy, and market presence. Forrester intends to update vendor evaluations regularly as product 
capabilities and vendor strategies evolve. 

ENDNOTES 

1 The customer now discovers researches, buys, and receives services across many touchpoints. Inconsistent 
assortments, content, promotions, pricing, policies, and capabilities will mean you are either intentionally 
frustrating the customer or just becoming obsolete. See the October 19, 201 1, "The Agile Commerce 
Platform" report. 

2 Online testing technology supports the mechanical aspects of online testing such as data collection, the 
execution of experiments, and generating reports. But online testing programs rely heavily on the time and 
skills of human participants to plan, design, and analyze a pipeline of experiments. See the August 10, 2010, 

"Organizing For Site Optimization" report. 

3 To close the gap between web analytics and comprehensive analytics for digital marketing, customer 
intelligence (CI) professionals must adopt an updated approach. Forrester calls this "digital intelligence," 
denned as "the capture, management, and analysis of data to provide a holistic view of the digital customer 
experience that drives the measurement, optimization, and execution of marketing tactics and business 
strategies." See the February 10, 2012, "Welcome To The Era Of Digital Intelligence" report. 

4 Net Promoter, Net Promoter Score, and NPS are registered trademarks of Bain, Fred Reichheld, and 
Satmetrix Systems. Source: Bain (http://www.bain.com/search.aspx?q=Net+Promoter); Satmetrix Systems 
(http ://www. satmetrix. com/) . 

5 Forrester's first online testing platform Wave was published just over two years ago. See the August 13, 2010, 
"The Forrester Wave ": Online Testing, Q3 2010" report. 



© 2013, Forrester Research, Inc. Reproduction Prohibited 



February 7, 2013 



About Forrester 



Global marketing and strategy leaders turn to Forrester to help 
them make the tough decisions necessary to capitalize on shifts 
in marketing, technology, and consumer behavior. We ensure your 
success by providing: 

■ Data-driven insight to understand the impact of changing 
consumer behavior. 

■ Forward-looking research and analysis to guide your decisions. 

■ Objective advice on tools and technologies to connect you with 
customers. 

■ Best practices for marketing and cross-channel strategy. 



FOR MORE INFORMATION 

To find out how Forrester Research can help you be successful every day please 
contact the office nearest you, or visit us at www.forrester.com. For a complete list 
of worldwide locations, visit www.forrester.com/about. 

CLIENT SUPPORT 

For information on hard-copy or electronic reprints, please contact Client Support 
at +1 866.367.7378, +1 617.613.5730, or clientsupport@forrester.com. We offer 
quantity discounts and special pricing for academic and nonprofit institutions. 



Forrester Focuses On 

Customer Intelligence Professionals 

In addition to integrating and managing complex data from initiatives 
such as customer segmentation, loyalty programs, and Web and mobile 
analytics, you must deliver multidimensional customer insights that 
power every marketing decision. Forrester's subject-matter expertise 
and deep understanding of your role will help you create forward- 
thinking strategies; weigh opportunity against risk; justify decisions; and 
optimize your individual, team, and corporate performance. 

« CHARLES IBRAHIM, client persona representing Customer Intelligence Professionals 



Forrester Research, Inc. (Nasdaq: FORR) is an independent research company that provides pragmatic and forward-thinking advice to 
global leaders in business and technology. Forrester works with professionals in 17 key roles at major companies providing proprietary 
research, customer insight, consulting, events, and peer-to-peer executive programs. For more than 29 years, Forrester has been making 
IT, marketing, and technology industry leaders successful every day. For more information, visit www.forrester.com. 89901 




